A moving picture consists of a plurality of pictures, and each picture consists of a predetermined number of pixels. Coding of a moving picture is carried out for each picture, and coding of each picture is carried out with taking a block which is obtained by partitioning the picture as a unit.
Generally, coding of a moving picture carries out compression of information amount by eliminating redundancy in time direction and in space direction.
For example, an inter picture prediction coding aiming at eliminating temporal redundancy carries out motion estimation and motion compensation for the target picture which is to be coded in block units with referring to pictures which are located temporally forward or backward of the target picture to generate prediction information, and then codes the difference between the prediction information and the target picture information. A picture which is located temporally forward of the target picture is a preceding picture, which is displayed earlier than the target picture, while a picture which is located temporally backward of the target picture is a following picture, which is displayed later than the target picture.
The H.264 coding system which is now being standardized as a moving picture coding system carries out motion compensation with respect to the target picture with referring to arbitrary two pictures which are located temporally forward and backward of the target picture. The H.264 is numbered as a recommendation number by ITU-T (International Telecommunication Union-Telecommunication Standardization Sector).
The coding aiming at eliminating spatial redundancy is carried out using coding information of the coded blocks which were already coded and which are positioned surrounding the target block.
However, when designing a coding apparatus and a decoding apparatus according to the H.264 system, there arises a problem that it is not possible to appropriately decide the sizes of the memory areas which are required in these apparatus. This is caused by that the degree of freedom concerning the number of reference pictures which are referred at inter picture prediction coding is extremely high.
Generally, in the coding method for a moving picture which consists of plural pictures, coding of a moving picture is carried out for each picture, and coding of each picture is carried out with taking a block which is obtained by dividing the picture (hereinafter, referred to as macroblock) as a unit.
For example, the coding of a picture P is carried out for each macroblock MB constituting the picture P, successively along an arrow B as shown in FIG. 24(a). Further, coding of the macroblock MB12 as a target block is carried out by predicting information such as motion vector, i.e., coding information, for the target macroblock with referring to the coding information for macroblocks MB3, MB4, MB5, and MB12 which are located at above and at sides of the target macroblock MB12.
Therefore, it is necessary to hold the coding information of such as the above macroblocks MB3 to MB11 which are located at above and at side of the target macroblock MB12 in order to perform prediction of coding information in coding macroblocks which are coded lately relative to the macroblock 12 until the coding process for the target macroblock MB12 is completed. In other words, in coding processing for each picture, coded information of macroblocks of approximately one row are always held. Therefore, in a case where the target image is an image that is laterally long, i.e., long in the horizontal direction, the coded information which is to be held in the coding processing becomes large. Accordingly, when designing an apparatus which can perform coding of an image that is laterally long, i.e., long in the horizontal direction, it is required to secure a large memory area for storing the coding information. Meanwhile, the coding information of macroblocks MB1 and MB2 shown in FIG. 24(b) are not referred to at coding of a macroblock whose coding order follows the macroblock MB12.
It is needless to say that a larger storage area for storing the coding information, is required also when designing a decoding apparatus that can decode an image having a longer lateral side, similarly as when designing a coding apparatus.
In the H.264 coding system, however, no requirements are currently imposed on images for their pixel numbers in the horizontal and vertical direction. This leads to a situation where the minimum size for a storage area that is required for appropriately performing coding or decoding of an image according to the H.264 standard is not decided.
In addition, when inter picture prediction coding or decoding is performed according to the H.264 standard, it is required to store all preceding and following pictures which are likely to be referred to in a picture memory.
To be brief, in the conventional MPEG (Moving Picture Experts Group)-2 or MPEG-4 method, pictures which can be referred to at the inter picture prediction coding or inter picture prediction decoding, i.e., reference candidate picture, are shared by all blocks of the target picture to be coded or decoded. For example, when the target picture is a B picture which can refer to two other processed pictures at maximum, pictures which are referred to at coding or decoding of each block, i.e., reference pictures, are selected from two reference candidate pictures that are decided for the target picture. When the target picture is a P picture which can refer to only one processed picture, only a reference picture which is decided for the target picture, is used as a picture that is referred to at coding or decoding of each block.
Descriptions of the reference pictures are given in a following part of the MPEG-4 Standard:
ISO/IEC 14496-2:1999(E)
Information technology—coding of audio-visual objects
Part 2: Visual
(1999-12-01)
PP. 328, 329 Annex N
On the other hand, in the H.264 coding method, a picture that is referred to at the inter picture prediction coding or inter picture prediction decoding, i.e., a reference picture, is selected from plural processed pictures whose image data are stored in a picture memory, for each block in a target picture to be coded or decoded. For example, in a case where the target picture is a B picture, for which each block is coded or decoded with referring to two other processed pictures at maximum, two pictures at maximum which are selected from plural processed pictures image data of which are stored in a picture memory, are referred to at coding or decoding of each block as reference pictures. In a case where the target picture is a P picture, for which each block is coded or decoded with referring to only one processed picture, a picture which is selected from plural processed pictures image data of which are stored in the picture memory is employed as a picture to be referred to at coding or decoding of each block, as a reference picture.
As discussed above, according to the MPEG-2 or MPEG-4 method, when the target picture is a P picture, one processed picture is a candidate for the reference picture, i.e., a reference candidate picture, and when the target picture is a B picture, two processed pictures are reference candidate pictures, while according to the H.264 method, both in cases where the target picture is a P picture and where the target picture is a B picture, plural processed pictures whose image data are stored in the picture memory, serve as candidates for the reference picture, i.e., reference candidate pictures.
FIG. 25 is a diagram for explaining a specific control of a picture memory according to the H.264 system. FIG. 25 shows a case where a picture memory can hold image data of four pictures. That is, in this case, three pictures among these four pictures, other than a target picture to be processed, are reference candidate pictures that may be referred to.
For example, when inter picture prediction coding or inter picture prediction decoding is carried out with taking picture P5 as a target picture to be processed as shown in FIG. 25, pictures P2 to P4 whose image data are stored in a picture memory Mpt are adopted as reference candidate pictures. Here, pictures P1 to P5 are arranged in coding order (or decoding order), and these pictures are coded or decoded in the order of arrangement. Thus, pictures which are displayed earlier than the target picture P5 among the reference candidate pictures P2 to P4, are preceding pictures for the target picture P5, and pictures which are displayed later than the target picture P5 among the reference candidate pictures P2 to P4, are following pictures for the target picture P5.
In a decoding apparatus, even when a previously decoded picture is a decoded picture that is other than reference candidate pictures which are employed for referring in the inter picture prediction decoding for the target picture, i.e., a decoded picture that is not employed as a reference picture, image data of the decoded picture must be stored in a picture memory as a display waiting picture until the order of display comes.
FIG. 26 are diagrams schematically showing the display wait pictures. FIG. 26(a) shows plural pictures constituting a moving picture, which are pictures that may be used as reference pictures (reference candidate pictures) [used], and pictures that would not be used as reference pictures [unused]. FIG. 26(b) shows the relationship between the decoding timing and the display timing of each picture.
For the purpose of illustration, FIG. 26 show a case where reference pictures [used] which are used at coding or decoding of each block in B picture are both or either of two reference candidate pictures that are commonly used for all blocks in this B picture, and reference pictures [used] which are used at the coding or decoding of each block in P picture are one reference candidate picture that is commonly used for all blocks in this P picture. However, in the H.264 method, a picture to be referred to at picture coding or decoding, i.e., a reference picture, is selected from plural processed pictures whose image data are stored in picture memory, for each block of a target picture to be coded or decoded. Therefore, the reference candidate pictures which are used at coding or decoding of B picture are not restricted to two specific pictures for each B picture as shown in FIG. 26(a), and the reference candidate picture that is used at coding or decoding of P picture is not restricted to one specific picture for each P picture as shown in FIG. 26(a).
In FIG. 26(a), reference candidate pictures for B picture B1 are an I picture I0 and B picture B2, and reference candidate pictures for B picture B2 are I picture I0 and P picture P4. Further, reference candidate pictures for B picture B3 are B picture B2 and P picture P4, and a reference candidate picture for P picture P8 is P picture P4.
FIG. 26(b) shows that respective pictures shown in FIG. 26(a) are decoded in the order of pictures I0, P4, B2, B1, B3, P8, B6, B5, B7, and then displayed in the order of pictures I0, B1, B2, B3, P4, B5, B6, B7, P8.
In FIG. 26(b), Tdec shows the time axis for representing the decoding times of respective pictures, and Tdsp shows the time axis for representing the display times of respective pictures. In addition, tdec(0), tdec(1), tdec(2), tdec(3), tdec(4), tdec(5), tdec(6), tdec(7), and tdec(8) denote the time periods in which decoding of pictures I0, B1, B2, B3, P4, B5, B6, B7, and P8 are carried out, respectively. Further, tdsp(0), tdsp(1), tdsp(2), tdsp(3), tdsp(4), tdsp(5), tdsp(6), tdsp(7), and tdsp(8) denote the time periods in which display of pictures I0, B1, B2, B3, P4, B5, B6, B7, and P8 are carried out, respectively.
In this case, the decoding times of the pictures B2, B1, B3, P8, B6, B5 and B7 roughly coincide with the display times of the pictures I0, B1, B2, B3, P4, B5, and B6, respectively, as shown in FIG. 26(b). In addition, the control of the picture memory is performed so that image data of decoded pictures are stored in the picture memory, and the image data of the pictures are deleted successively from pictures which have been displayed.
In this case, for example, I picture I0 is displayed after waiting for the B picture B2 being decoded.
The image data of the reference candidate pictures are stored in the picture memory, and these pictures are employed as reference pictures until these pictures are displayed later. However, image data of pictures that are not referred to also must be kept until these pictures are displayed after being decoded. The image data of such decoded pictures that are not employed as reference pictures, are stored in a predetermined memory as display waiting pictures until these pictures are displayed.
In FIG. 26(a), B picture B1 that is decoded after I picture I0 being decoded, is a picture that is not used as a reference picture and will be displayed subsequent to I picture I0, and therefore, it can be displayed immediately after the decoding thereof, while B picture B3 that is decoded after B picture B1 being decoded, is a picture that will be displayed after B picture B2 being displayed, that follows B picture B1, and therefore, it will be displayed with waiting for a display period of one picture, i.e., B picture B2, after B picture B1 being decoded.
In this case, the number of display waiting picture(s) at the decoding start timing of P picture P8 is only one, i.e., B picture B3.
Further, there arises no problem by that the image data of pictures which will not be used as reference pictures should be deleted immediately after the display of those pictures is completed. However, the timing of deleting the image data of these pictures may be other than immediately after the display of those pictures being performed.
In such cases, the image data of the pictures which are not used as reference pictures, would be kept stored in the memory also after these pictures are displayed, and these pictures whose image data are held in the picture memory are also handled as display waiting pictures.
For example, in a case where the control of the picture memory is carried out such that image data of a picture which is stored in the picture memory but is not used as a reference picture, is deleted from the picture memory after a display time of one picture expiring after the picture was displayed, the number of display waiting pictures at the start of decoding of P picture P8 amounts to two, i.e., B picture B2 and B picture B3.
As discussed above, plural decoded or coded pictures are stored in a picture memory of a decoding apparatus or a coding apparatus, while in the H.264 coding system there is currently no limitations on the maximum number of reference candidate pictures which are employed at the inter picture prediction coding or decoding, i.e., the maximum reference picture number.
Thus, in the design of a coding apparatus or a decoding apparatus according to the H.264 system, the maximum number of decoded or coded pictures which are to be stored in the picture memory cannot be established, and therefore the capacity of a storage area to be mounted in the apparatus cannot be decided.
The present invention is made to solve the above-mentioned problems, and has for its object to provide a moving picture coding method and a moving picture decoding method which can correctly judge the capability/incapability of coding or decoding for a moving picture as well as efficiently utilize a memory area mounted in a coding/decoding apparatus, and thereby enables designing a memory area in a coding/decoding apparatus according to a coding method which imposes no limitations on the capacity of the memory.